Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks

Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks, where now excessively large models are used. However, such face several problems during the learning process, mainly due to redundancy individual neurons, which results in sub-optimal accuracy or need additional training steps. Here, we explore diversity neurons within hidden layer and analyze how affects predictions model. As following, introduce techniques dynamically reinforce between training. These decorrelation improve at early stages occasionally help overcome local minima faster. Additionally, describe novel weight initialization method obtain decorrelated, yet stochastic fast efficient network Decorrelated our case shows about 40% relative increase test first 5 epochs.